Exploring L2A Reflectance¶

Summary

In this notebook we will open a Level 2A (L2A) Reflectance product file (netcdf) from Earth Surface Minteral Dust Source Investigation (EMIT). We will inspect the structure and plot the spectra of individual pixels and spatial coverage of a single band. Next, we will orthorectify the imagery using the included geometry lookup table (GLT). Finally, we will take advantage of the holoviews streams module to build an interactive plot.

Background

The EMIT instrument is an imaging spectrometer that measures light in visible and infrared wavelengths. These measurements display unique spectral signatures that correspond to the composition on the Earth's surface. The EMIT mission focuses specifically on mapping the composition of minerals to better understand the effects of mineral dust throughout the Earth system and human populations now and in the future. More details about EMIT and its associated products can be found in the README.md and on the EMIT website.

The L2A Reflectance Product contains estimated surface reflectance. Surface reflectance is the fraction of incoming solar radiation reflected Earth's surface. Materials reflect proportions of radiation differently based upon their chemical composition. This means that reflectance information can be used to determine the composition of a target. In this guide you will learn how to plot a band from the L2A reflectance spatially and look at the spectral curve associated with individual pixels.

Requirements

  • Set up Python Environment - See setup_instructions.md in the /setup/ folder
  • Download the required EMIT data - See setup_instructions.md in the /setup/ folder

Learning Objectives

  • How to open an EMIT .nc file as an xarray.Dataset
  • Apply the Geometry Lookup Table (GLT) to orthorectify the image.
  • How to plot the spectra of pixels
  • How to plot specific bands as images
  • How to make an interactive plot to visualize spectra

Tutorial Outline

1.1 Setup
1.2 Opening The Data
1.3 Plotting Data - Non-Orthorectified
1.4 Orthorectification
1.5 Plotting Data - Orthorectified
1.6 Saving Orthorectified Data
1.7 Interactive Plots

Interactive Plot

1.1 Setup¶

Import the required Python libraries.

In [1]:
from osgeo import gdal
import numpy as np
import math
import rasterio as rio
import xarray as xr
import holoviews as hv
import hvplot.xarray
import netCDF4 as nc

For the workshop, we will use the EMIT granule that can be downloaded here. If you are on the provided cloud computing environment, it has already been downloaded and can be found at the filepath below.

In [2]:
fp = '/home/jovyan/shared/2023-emit-tutorials/data/EMIT_L2A_RFL_001_20220903T163129_2224611_012.nc'

1.2 Opening EMIT Data¶

EMIT L2A Reflectance Data are distributed in a non-orthocorrected spatially raw NetCDF4 (.nc) format consisting of the data and its associated metadata. Inside the L2A Reflectance .nc file there are 3 groups. Groups can be thought of as containers to organize the data.

  1. The root group that can be considered the main dataset contains the reflectance data described by the downtrack, crosstrack, and bands dimensions.
  2. The sensor_band_parameters group containing the wavelength center and the full-width half maximum (FWHM) of each band.
  3. The location group contains latitude and longitude values at the center of each pixel described by the crosstrack and downtrack dimensions, as well as a geometry lookup table (GLT) described by the ortho_x and ortho_y dimensions. The GLT is an orthorectified image (EPSG:4326) consisting of 2 layers containing downtrack and crosstrack indices. These index positions allow us to quickly project the raw data onto this geographic grid.

To access the .nc file we will use the netCDF4 and xarray libraries. The netCDF4 library will be used to explore thee data structure, then we will use xarray to work with the data. xarray is a python package for working with labelled multi-dimensional arrays. It provides a data model where data, dimensions, and attributes together in an easily interpretable way.

In [3]:
ds = nc.Dataset(fp)
ds
Out[3]:
<class 'netCDF4._netCDF4.Dataset'>
root group (NETCDF4 data model, file format HDF5):
    ncei_template_version: NCEI_NetCDF_Swath_Template_v2.0
    summary: The Earth Surface Mineral Dust Source Investigation (EMIT) is an Earth Ventures-Instrument (EVI-4) Mission that maps the surface mineralogy of arid dust source regions via imaging spectroscopy in the visible and short-wave infrared (VSWIR). Installed on the International Space Station (ISS), the EMIT instrument is a Dyson imaging spectrometer that uses contiguous spectroscopic measurements from 410 to 2450 nm to resolve absoprtion features of iron oxides, clays, sulfates, carbonates, and other dust-forming minerals. During its one-year mission, EMIT will observe the sunlit Earth's dust source regions that occur within +/-52° latitude and produce maps of the source regions that can be used to improve forecasts of the role of mineral dust in the radiative forcing (warming or cooling) of the atmosphere.\n\nThis file contains L2A estimated surface reflectances         and geolocation data. Reflectance estimates are created using an Optimal Estimation technique - see ATBD for          details. Reflectance values are reported as fractions (relative to 1). 
    keywords: Imaging Spectroscopy, minerals, EMIT, dust, radiative forcing
    Conventions: CF-1.63
    sensor: EMIT (Earth Surface Mineral Dust Source Investigation)
    instrument: EMIT
    platform: ISS
    processing_version: V1.0
    institution: NASA Jet Propulsion Laboratory/California Institute of Technology
    license: https://science.nasa.gov/earth-science/earth-science-data/data-information-policy/
    naming_authority: LPDAAC
    date_created: 2022-11-14T09:50:54Z
    keywords_vocabulary: NASA Global Change Master Directory (GCMD) Science Keywords
    stdname_vocabulary: NetCDF Climate and Forecast (CF) Metadata Convention
    creator_name: Jet Propulsion Laboratory/California Institute of Technology
    creator_email: sarah.r.lundeen@jpl.nasa.gov
    creator_url: https://earth.jpl.nasa.gov/emit/
    project: Earth Surface Mineral Dust Source Investigation
    project_url: https://emit.jpl.nasa.gov/
    publisher_name: USGS LPDAAC
    publisher_url: https://lpdaac.usgs.gov
    publisher_email: lpdaac@usgs.gov
    identifier_product_doi_authority: https://doi.org
    flight_line: emit20220903t163129_o24611_s000
    time_coverage_start: 2022-09-03T16:31:29+0000
    time_coverage_end: 2022-09-03T16:31:41+0000
    software_build_version: 010603
    product_version: 01
    history: PGE Input files: radiance_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_rdn_b0106_v01.img, pixel_locations_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_loc_b0106_v01.img, observation_parameters_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_obs_b0106_v01.img, surface_model_config=/beegfs/store/emit/ops/repos/emit-sds-l2a/surface/surface_20221020.json
    easternmost_longitude: -62.5120945327963
    northernmost_latitude: -39.3067591475017
    westernmost_longitude: -61.236221412633064
    southernmost_latitude: -40.39610428069674
    spatialResolution: 0.000542232520256367
    spatial_ref: GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
    geotransform: [-6.25120945e+01  5.42232520e-04 -0.00000000e+00 -3.93067591e+01
 -0.00000000e+00 -5.42232520e-04]
    day_night_flag: Day
    title: EMIT L2A Surface Reflectance 60 m V001
    dimensions(sizes): downtrack(1280), crosstrack(1242), bands(285), ortho_y(2009), ortho_x(2353)
    variables(dimensions): float32 reflectance(downtrack, crosstrack, bands)
    groups: sensor_band_parameters, location

From this output, we can see the reflectance variable, and the sensor_band_parameters and location groups. We can also see the dimensions, their sizes, and file metadata.

Now that we have a better understanding of the structure of the file, read the EMIT data as an xarray.Dataset and preview it.

In [4]:
refl = xr.open_dataset(fp)
refl
Out[4]:
<xarray.Dataset>
Dimensions:      (downtrack: 1280, crosstrack: 1242, bands: 285)
Dimensions without coordinates: downtrack, crosstrack, bands
Data variables:
    reflectance  (downtrack, crosstrack, bands) float32 ...
Attributes: (12/38)
    ncei_template_version:             NCEI_NetCDF_Swath_Template_v2.0
    summary:                           The Earth Surface Mineral Dust Source ...
    keywords:                          Imaging Spectroscopy, minerals, EMIT, ...
    Conventions:                       CF-1.63
    sensor:                            EMIT (Earth Surface Mineral Dust Sourc...
    instrument:                        EMIT
    ...                                ...
    southernmost_latitude:             -40.39610428069674
    spatialResolution:                 0.000542232520256367
    spatial_ref:                       GEOGCS["WGS 84",DATUM["WGS_1984",SPHER...
    geotransform:                      [-6.25120945e+01  5.42232520e-04 -0.00...
    day_night_flag:                    Day
    title:                             EMIT L2A Surface Reflectance 60 m V001
xarray.Dataset
    • downtrack: 1280
    • crosstrack: 1242
    • bands: 285
      • reflectance
        (downtrack, crosstrack, bands)
        float32
        ...
        long_name :
        Surface Reflectance
        units :
        unitless
        [453081600 values with dtype=float32]
      • ncei_template_version :
        NCEI_NetCDF_Swath_Template_v2.0
        summary :
        The Earth Surface Mineral Dust Source Investigation (EMIT) is an Earth Ventures-Instrument (EVI-4) Mission that maps the surface mineralogy of arid dust source regions via imaging spectroscopy in the visible and short-wave infrared (VSWIR). Installed on the International Space Station (ISS), the EMIT instrument is a Dyson imaging spectrometer that uses contiguous spectroscopic measurements from 410 to 2450 nm to resolve absoprtion features of iron oxides, clays, sulfates, carbonates, and other dust-forming minerals. During its one-year mission, EMIT will observe the sunlit Earth's dust source regions that occur within +/-52° latitude and produce maps of the source regions that can be used to improve forecasts of the role of mineral dust in the radiative forcing (warming or cooling) of the atmosphere.\n\nThis file contains L2A estimated surface reflectances and geolocation data. Reflectance estimates are created using an Optimal Estimation technique - see ATBD for details. Reflectance values are reported as fractions (relative to 1).
        keywords :
        Imaging Spectroscopy, minerals, EMIT, dust, radiative forcing
        Conventions :
        CF-1.63
        sensor :
        EMIT (Earth Surface Mineral Dust Source Investigation)
        instrument :
        EMIT
        platform :
        ISS
        processing_version :
        V1.0
        institution :
        NASA Jet Propulsion Laboratory/California Institute of Technology
        license :
        https://science.nasa.gov/earth-science/earth-science-data/data-information-policy/
        naming_authority :
        LPDAAC
        date_created :
        2022-11-14T09:50:54Z
        keywords_vocabulary :
        NASA Global Change Master Directory (GCMD) Science Keywords
        stdname_vocabulary :
        NetCDF Climate and Forecast (CF) Metadata Convention
        creator_name :
        Jet Propulsion Laboratory/California Institute of Technology
        creator_email :
        sarah.r.lundeen@jpl.nasa.gov
        creator_url :
        https://earth.jpl.nasa.gov/emit/
        project :
        Earth Surface Mineral Dust Source Investigation
        project_url :
        https://emit.jpl.nasa.gov/
        publisher_name :
        USGS LPDAAC
        publisher_url :
        https://lpdaac.usgs.gov
        publisher_email :
        lpdaac@usgs.gov
        identifier_product_doi_authority :
        https://doi.org
        flight_line :
        emit20220903t163129_o24611_s000
        time_coverage_start :
        2022-09-03T16:31:29+0000
        time_coverage_end :
        2022-09-03T16:31:41+0000
        software_build_version :
        010603
        product_version :
        01
        history :
        PGE Input files: radiance_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_rdn_b0106_v01.img, pixel_locations_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_loc_b0106_v01.img, observation_parameters_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_obs_b0106_v01.img, surface_model_config=/beegfs/store/emit/ops/repos/emit-sds-l2a/surface/surface_20221020.json
        easternmost_longitude :
        -62.5120945327963
        northernmost_latitude :
        -39.3067591475017
        westernmost_longitude :
        -61.236221412633064
        southernmost_latitude :
        -40.39610428069674
        spatialResolution :
        0.000542232520256367
        spatial_ref :
        GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
        geotransform :
        [-6.25120945e+01 5.42232520e-04 -0.00000000e+00 -3.93067591e+01 -0.00000000e+00 -5.42232520e-04]
        day_night_flag :
        Day
        title :
        EMIT L2A Surface Reflectance 60 m V001

      This xarray dataset only contains the reflectance variable and attributes metadata, not the data from the other groups in the file. This is because xarray only supports reading non-hierarchical (flat) datasets, meaning that when loading a NetCDF into an xarray.Dataset, only the root group is added. The other groups will have to be read into xarray separately. We can list them using the netCDF4 library to get the group names, then use that to add them to new xarray datasets.

      In [5]:
      ds.groups.keys()
      Out[5]:
      dict_keys(['sensor_band_parameters', 'location'])

      Now that we know the other group names, read the sensor_band_parameters and location groups into their own xarray datasets.

      In [6]:
      wvl = xr.open_dataset(fp,group='sensor_band_parameters')
wvl
      Out[6]:
      <xarray.Dataset>
Dimensions:      (bands: 285)
Dimensions without coordinates: bands
Data variables:
    wavelengths  (bands) float32 ...
    fwhm         (bands) float32 ...
      xarray.Dataset
        • bands: 285
          • wavelengths
            (bands)
            float32
            ...
            long_name :
            Wavelength Centers
            units :
            nm
            [285 values with dtype=float32]
          • fwhm
            (bands)
            float32
            ...
            long_name :
            Full Width at Half Max
            units :
            nm
            [285 values with dtype=float32]
          In [7]:
          loc = xr.open_dataset(fp,group='location')
loc
          Out[7]:
          <xarray.Dataset>
Dimensions:  (downtrack: 1280, crosstrack: 1242, ortho_y: 2009, ortho_x: 2353)
Dimensions without coordinates: downtrack, crosstrack, ortho_y, ortho_x
Data variables:
    lat      (downtrack, crosstrack) float64 ...
    lon      (downtrack, crosstrack) float64 ...
    elev     (downtrack, crosstrack) float64 ...
    glt_x    (ortho_y, ortho_x) float64 ...
    glt_y    (ortho_y, ortho_x) float64 ...
          xarray.Dataset
            • downtrack: 1280
            • crosstrack: 1242
            • ortho_y: 2009
            • ortho_x: 2353
              • lat
                (downtrack, crosstrack)
                float64
                ...
                long_name :
                Longitude (WGS-84)
                units :
                degrees east
                [1589760 values with dtype=float64]
              • lon
                (downtrack, crosstrack)
                float64
                ...
                long_name :
                Latitude (WGS-84)
                units :
                degrees north
                [1589760 values with dtype=float64]
              • elev
                (downtrack, crosstrack)
                float64
                ...
                long_name :
                Surface Elevation
                units :
                m
                [1589760 values with dtype=float64]
              • glt_x
                (ortho_y, ortho_x)
                float64
                ...
                long_name :
                GLT Sample Lookup
                units :
                pixel location
                [4727177 values with dtype=float64]
              • glt_y
                (ortho_y, ortho_x)
                float64
                ...
                long_name :
                GLT Line Lookup
                units :
                pixel location
                [4727177 values with dtype=float64]

              Merge the reflectance and location datasets into a single xarray.Dataset.

              In [8]:
              ds = xr.merge([refl, loc])
ds
              Out[8]:
              <xarray.Dataset>
Dimensions:      (downtrack: 1280, crosstrack: 1242, bands: 285, ortho_y: 2009,
                  ortho_x: 2353)
Dimensions without coordinates: downtrack, crosstrack, bands, ortho_y, ortho_x
Data variables:
    reflectance  (downtrack, crosstrack, bands) float32 ...
    lat          (downtrack, crosstrack) float64 ...
    lon          (downtrack, crosstrack) float64 ...
    elev         (downtrack, crosstrack) float64 ...
    glt_x        (ortho_y, ortho_x) float64 ...
    glt_y        (ortho_y, ortho_x) float64 ...
Attributes: (12/38)
    ncei_template_version:             NCEI_NetCDF_Swath_Template_v2.0
    summary:                           The Earth Surface Mineral Dust Source ...
    keywords:                          Imaging Spectroscopy, minerals, EMIT, ...
    Conventions:                       CF-1.63
    sensor:                            EMIT (Earth Surface Mineral Dust Sourc...
    instrument:                        EMIT
    ...                                ...
    southernmost_latitude:             -40.39610428069674
    spatialResolution:                 0.000542232520256367
    spatial_ref:                       GEOGCS["WGS 84",DATUM["WGS_1984",SPHER...
    geotransform:                      [-6.25120945e+01  5.42232520e-04 -0.00...
    day_night_flag:                    Day
    title:                             EMIT L2A Surface Reflectance 60 m V001
              xarray.Dataset
                • downtrack: 1280
                • crosstrack: 1242
                • bands: 285
                • ortho_y: 2009
                • ortho_x: 2353
                  • reflectance
                    (downtrack, crosstrack, bands)
                    float32
                    ...
                    long_name :
                    Surface Reflectance
                    units :
                    unitless
                    [453081600 values with dtype=float32]
                  • lat
                    (downtrack, crosstrack)
                    float64
                    ...
                    long_name :
                    Longitude (WGS-84)
                    units :
                    degrees east
                    [1589760 values with dtype=float64]
                  • lon
                    (downtrack, crosstrack)
                    float64
                    ...
                    long_name :
                    Latitude (WGS-84)
                    units :
                    degrees north
                    [1589760 values with dtype=float64]
                  • elev
                    (downtrack, crosstrack)
                    float64
                    ...
                    long_name :
                    Surface Elevation
                    units :
                    m
                    [1589760 values with dtype=float64]
                  • glt_x
                    (ortho_y, ortho_x)
                    float64
                    ...
                    long_name :
                    GLT Sample Lookup
                    units :
                    pixel location
                    [4727177 values with dtype=float64]
                  • glt_y
                    (ortho_y, ortho_x)
                    float64
                    ...
                    long_name :
                    GLT Line Lookup
                    units :
                    pixel location
                    [4727177 values with dtype=float64]
                  • ncei_template_version :
                    NCEI_NetCDF_Swath_Template_v2.0
                    summary :
                    The Earth Surface Mineral Dust Source Investigation (EMIT) is an Earth Ventures-Instrument (EVI-4) Mission that maps the surface mineralogy of arid dust source regions via imaging spectroscopy in the visible and short-wave infrared (VSWIR). Installed on the International Space Station (ISS), the EMIT instrument is a Dyson imaging spectrometer that uses contiguous spectroscopic measurements from 410 to 2450 nm to resolve absoprtion features of iron oxides, clays, sulfates, carbonates, and other dust-forming minerals. During its one-year mission, EMIT will observe the sunlit Earth's dust source regions that occur within +/-52° latitude and produce maps of the source regions that can be used to improve forecasts of the role of mineral dust in the radiative forcing (warming or cooling) of the atmosphere.\n\nThis file contains L2A estimated surface reflectances and geolocation data. Reflectance estimates are created using an Optimal Estimation technique - see ATBD for details. Reflectance values are reported as fractions (relative to 1).
                    keywords :
                    Imaging Spectroscopy, minerals, EMIT, dust, radiative forcing
                    Conventions :
                    CF-1.63
                    sensor :
                    EMIT (Earth Surface Mineral Dust Source Investigation)
                    instrument :
                    EMIT
                    platform :
                    ISS
                    processing_version :
                    V1.0
                    institution :
                    NASA Jet Propulsion Laboratory/California Institute of Technology
                    license :
                    https://science.nasa.gov/earth-science/earth-science-data/data-information-policy/
                    naming_authority :
                    LPDAAC
                    date_created :
                    2022-11-14T09:50:54Z
                    keywords_vocabulary :
                    NASA Global Change Master Directory (GCMD) Science Keywords
                    stdname_vocabulary :
                    NetCDF Climate and Forecast (CF) Metadata Convention
                    creator_name :
                    Jet Propulsion Laboratory/California Institute of Technology
                    creator_email :
                    sarah.r.lundeen@jpl.nasa.gov
                    creator_url :
                    https://earth.jpl.nasa.gov/emit/
                    project :
                    Earth Surface Mineral Dust Source Investigation
                    project_url :
                    https://emit.jpl.nasa.gov/
                    publisher_name :
                    USGS LPDAAC
                    publisher_url :
                    https://lpdaac.usgs.gov
                    publisher_email :
                    lpdaac@usgs.gov
                    identifier_product_doi_authority :
                    https://doi.org
                    flight_line :
                    emit20220903t163129_o24611_s000
                    time_coverage_start :
                    2022-09-03T16:31:29+0000
                    time_coverage_end :
                    2022-09-03T16:31:41+0000
                    software_build_version :
                    010603
                    product_version :
                    01
                    history :
                    PGE Input files: radiance_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_rdn_b0106_v01.img, pixel_locations_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_loc_b0106_v01.img, observation_parameters_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_obs_b0106_v01.img, surface_model_config=/beegfs/store/emit/ops/repos/emit-sds-l2a/surface/surface_20221020.json
                    easternmost_longitude :
                    -62.5120945327963
                    northernmost_latitude :
                    -39.3067591475017
                    westernmost_longitude :
                    -61.236221412633064
                    southernmost_latitude :
                    -40.39610428069674
                    spatialResolution :
                    0.000542232520256367
                    spatial_ref :
                    GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
                    geotransform :
                    [-6.25120945e+01 5.42232520e-04 -0.00000000e+00 -3.93067591e+01 -0.00000000e+00 -5.42232520e-04]
                    day_night_flag :
                    Day
                    title :
                    EMIT L2A Surface Reflectance 60 m V001

                  Instead of also merging the sensor_band_parameters as variables, we can treat them as coordinates for the band dimension, along with downtrack and crosstrack to describe the spatially raw dimensions. This will allow us to utilize some additional features of xarray.

                  In [9]:
                  ds = ds.assign_coords({'downtrack':(['downtrack'], refl.downtrack.data),'crosstrack':(['crosstrack'],refl.crosstrack.data), **wvl.variables}) # This will utilize the wvl dataset dictionary as the ds coordinates dictionary
ds
                  Out[9]:
                  <xarray.Dataset>
Dimensions:      (downtrack: 1280, crosstrack: 1242, bands: 285, ortho_y: 2009,
                  ortho_x: 2353)
Coordinates:
  * downtrack    (downtrack) int64 0 1 2 3 4 5 ... 1274 1275 1276 1277 1278 1279
  * crosstrack   (crosstrack) int64 0 1 2 3 4 5 ... 1237 1238 1239 1240 1241
    wavelengths  (bands) float32 ...
    fwhm         (bands) float32 ...
Dimensions without coordinates: bands, ortho_y, ortho_x
Data variables:
    reflectance  (downtrack, crosstrack, bands) float32 ...
    lat          (downtrack, crosstrack) float64 ...
    lon          (downtrack, crosstrack) float64 ...
    elev         (downtrack, crosstrack) float64 ...
    glt_x        (ortho_y, ortho_x) float64 ...
    glt_y        (ortho_y, ortho_x) float64 ...
Attributes: (12/38)
    ncei_template_version:             NCEI_NetCDF_Swath_Template_v2.0
    summary:                           The Earth Surface Mineral Dust Source ...
    keywords:                          Imaging Spectroscopy, minerals, EMIT, ...
    Conventions:                       CF-1.63
    sensor:                            EMIT (Earth Surface Mineral Dust Sourc...
    instrument:                        EMIT
    ...                                ...
    southernmost_latitude:             -40.39610428069674
    spatialResolution:                 0.000542232520256367
    spatial_ref:                       GEOGCS["WGS 84",DATUM["WGS_1984",SPHER...
    geotransform:                      [-6.25120945e+01  5.42232520e-04 -0.00...
    day_night_flag:                    Day
    title:                             EMIT L2A Surface Reflectance 60 m V001
                  xarray.Dataset
                    • downtrack: 1280
                    • crosstrack: 1242
                    • bands: 285
                    • ortho_y: 2009
                    • ortho_x: 2353
                    • downtrack
                      (downtrack)
                      int64
                      0 1 2 3 4 ... 1276 1277 1278 1279
                      array([   0,    1,    2, ..., 1277, 1278, 1279])
                    • crosstrack
                      (crosstrack)
                      int64
                      0 1 2 3 4 ... 1238 1239 1240 1241
                      array([   0,    1,    2, ..., 1239, 1240, 1241])
                    • wavelengths
                      (bands)
                      float32
                      ...
                      long_name :
                      Wavelength Centers
                      units :
                      nm
                      [285 values with dtype=float32]
                    • fwhm
                      (bands)
                      float32
                      ...
                      long_name :
                      Full Width at Half Max
                      units :
                      nm
                      [285 values with dtype=float32]
                    • reflectance
                      (downtrack, crosstrack, bands)
                      float32
                      ...
                      long_name :
                      Surface Reflectance
                      units :
                      unitless
                      [453081600 values with dtype=float32]
                    • lat
                      (downtrack, crosstrack)
                      float64
                      ...
                      long_name :
                      Longitude (WGS-84)
                      units :
                      degrees east
                      [1589760 values with dtype=float64]
                    • lon
                      (downtrack, crosstrack)
                      float64
                      ...
                      long_name :
                      Latitude (WGS-84)
                      units :
                      degrees north
                      [1589760 values with dtype=float64]
                    • elev
                      (downtrack, crosstrack)
                      float64
                      ...
                      long_name :
                      Surface Elevation
                      units :
                      m
                      [1589760 values with dtype=float64]
                    • glt_x
                      (ortho_y, ortho_x)
                      float64
                      ...
                      long_name :
                      GLT Sample Lookup
                      units :
                      pixel location
                      [4727177 values with dtype=float64]
                    • glt_y
                      (ortho_y, ortho_x)
                      float64
                      ...
                      long_name :
                      GLT Line Lookup
                      units :
                      pixel location
                      [4727177 values with dtype=float64]
                    • downtrack
                      PandasIndex
                      PandasIndex(Int64Index([   0,    1,    2,    3,    4,    5,    6,    7,    8,    9,
            ...
            1270, 1271, 1272, 1273, 1274, 1275, 1276, 1277, 1278, 1279],
           dtype='int64', name='downtrack', length=1280))
                    • crosstrack
                      PandasIndex
                      PandasIndex(Int64Index([   0,    1,    2,    3,    4,    5,    6,    7,    8,    9,
            ...
            1232, 1233, 1234, 1235, 1236, 1237, 1238, 1239, 1240, 1241],
           dtype='int64', name='crosstrack', length=1242))
                  • ncei_template_version :
                    NCEI_NetCDF_Swath_Template_v2.0
                    summary :
                    The Earth Surface Mineral Dust Source Investigation (EMIT) is an Earth Ventures-Instrument (EVI-4) Mission that maps the surface mineralogy of arid dust source regions via imaging spectroscopy in the visible and short-wave infrared (VSWIR). Installed on the International Space Station (ISS), the EMIT instrument is a Dyson imaging spectrometer that uses contiguous spectroscopic measurements from 410 to 2450 nm to resolve absoprtion features of iron oxides, clays, sulfates, carbonates, and other dust-forming minerals. During its one-year mission, EMIT will observe the sunlit Earth's dust source regions that occur within +/-52° latitude and produce maps of the source regions that can be used to improve forecasts of the role of mineral dust in the radiative forcing (warming or cooling) of the atmosphere.\n\nThis file contains L2A estimated surface reflectances and geolocation data. Reflectance estimates are created using an Optimal Estimation technique - see ATBD for details. Reflectance values are reported as fractions (relative to 1).
                    keywords :
                    Imaging Spectroscopy, minerals, EMIT, dust, radiative forcing
                    Conventions :
                    CF-1.63
                    sensor :
                    EMIT (Earth Surface Mineral Dust Source Investigation)
                    instrument :
                    EMIT
                    platform :
                    ISS
                    processing_version :
                    V1.0
                    institution :
                    NASA Jet Propulsion Laboratory/California Institute of Technology
                    license :
                    https://science.nasa.gov/earth-science/earth-science-data/data-information-policy/
                    naming_authority :
                    LPDAAC
                    date_created :
                    2022-11-14T09:50:54Z
                    keywords_vocabulary :
                    NASA Global Change Master Directory (GCMD) Science Keywords
                    stdname_vocabulary :
                    NetCDF Climate and Forecast (CF) Metadata Convention
                    creator_name :
                    Jet Propulsion Laboratory/California Institute of Technology
                    creator_email :
                    sarah.r.lundeen@jpl.nasa.gov
                    creator_url :
                    https://earth.jpl.nasa.gov/emit/
                    project :
                    Earth Surface Mineral Dust Source Investigation
                    project_url :
                    https://emit.jpl.nasa.gov/
                    publisher_name :
                    USGS LPDAAC
                    publisher_url :
                    https://lpdaac.usgs.gov
                    publisher_email :
                    lpdaac@usgs.gov
                    identifier_product_doi_authority :
                    https://doi.org
                    flight_line :
                    emit20220903t163129_o24611_s000
                    time_coverage_start :
                    2022-09-03T16:31:29+0000
                    time_coverage_end :
                    2022-09-03T16:31:41+0000
                    software_build_version :
                    010603
                    product_version :
                    01
                    history :
                    PGE Input files: radiance_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_rdn_b0106_v01.img, pixel_locations_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_loc_b0106_v01.img, observation_parameters_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_obs_b0106_v01.img, surface_model_config=/beegfs/store/emit/ops/repos/emit-sds-l2a/surface/surface_20221020.json
                    easternmost_longitude :
                    -62.5120945327963
                    northernmost_latitude :
                    -39.3067591475017
                    westernmost_longitude :
                    -61.236221412633064
                    southernmost_latitude :
                    -40.39610428069674
                    spatialResolution :
                    0.000542232520256367
                    spatial_ref :
                    GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
                    geotransform :
                    [-6.25120945e+01 5.42232520e-04 -0.00000000e+00 -3.93067591e+01 -0.00000000e+00 -5.42232520e-04]
                    day_night_flag :
                    Day
                    title :
                    EMIT L2A Surface Reflectance 60 m V001

                  Now we have an xarray.Dataset containing all of the information from EMIT netCDF file. Since these datasets are large, we can go ahead and delete objects we won't be using to conserve memory.

                  In [10]:
                  del refl
del wvl
del loc

                  1.3 Visualizing Spectra - Non-Orthorectified¶

                  Pick a random downtrack and crosstrack location. Here we chose 660, 370 (downtrack,crosstrack). Next use the sel() function from xarray and the hvplot.line() functions to first select the spatial position and then plot a line showing the reflectance at that location.

                  In [11]:
                  example = ds['reflectance'].sel(downtrack=660,crosstrack=370)
example.hvplot.line(y='reflectance',x='wavelengths', color='black')
                  Out[11]:

                  We can see some flat regions in the spectral curve around 1320-1440 nm and 1770-1970 nm. These are where water absoption features in these regions were removed. Typically this data is noisy due to the moisture present in the atmosphere; therefore, these spectral regions offer little information about targets and can be excluded from calculations.

                  Although they have been assigned a value of -0.01, we can mask them to improve visualization, by assigning their value to np.nan.

                  In [12]:
                  ds['reflectance'].data[ds['reflectance'].data == -0.01] = np.nan
# In newer versions of the file, this line below that takes advantage of the `good_wavelengths` data in the `sensor_band_parameters` group.
#ds['reflectance'].data[ds['good_wavelengths'].data==0] = np.nan

                  Plot the filtered reflectance values using the same downtrack and crosstrack position as above.

                  In [13]:
                  ds['reflectance'].sel(downtrack=660,crosstrack=370).hvplot.line(y='reflectance',x='wavelengths', color='black')
                  Out[13]:

                  Without these data we can better interpret the spectral curve and hvplot will do a better job automatically scaling our axes.

                  We can also plot the data spatially. Find the band nearest the 850nm wavelength in the NIR, then plot the data spatially using the sel() function to select only that band and using hvplot.image() to view the reflectance at 850nm of each pixel across the acquired region.

                  In [14]:
                  b850 = np.nanargmin(abs(ds['wavelengths'].values-850)) # Find band nearest to value of 850 nm (NIR)
b850
                  Out[14]:
                  63
                  In [15]:
                  ds.sel(bands=b850).hvplot.image(cmap='viridis', aspect = 'equal', rasterize=True)
                  Out[15]:

                  1.4 Orthorectification¶

                  The 'real' orthorectifation process has already been done for EMIT data. Here we are using the crosstrack and downtrack indices contained in the GLT to place our spatially raw reflectance data a into geographic grid with the ortho_x and ortho_y dimensions. As previously mentioned a Geometry Lookup Table (GLT) is included in the location group of the netCDF4 file. Applying the GLT will orthorectify the data and give us Latitude and Longitude positional information.

                  Before using the GLT to orthorectify the data, examine the location group from the dataset by reading it into xarray.

                  In [16]:
                  loc = xr.open_dataset(fp,group='location')
loc
                  Out[16]:
                  <xarray.Dataset>
Dimensions:  (downtrack: 1280, crosstrack: 1242, ortho_y: 2009, ortho_x: 2353)
Dimensions without coordinates: downtrack, crosstrack, ortho_y, ortho_x
Data variables:
    lat      (downtrack, crosstrack) float64 ...
    lon      (downtrack, crosstrack) float64 ...
    elev     (downtrack, crosstrack) float64 ...
    glt_x    (ortho_y, ortho_x) float64 ...
    glt_y    (ortho_y, ortho_x) float64 ...
                  xarray.Dataset
                    • downtrack: 1280
                    • crosstrack: 1242
                    • ortho_y: 2009
                    • ortho_x: 2353
                      • lat
                        (downtrack, crosstrack)
                        float64
                        ...
                        long_name :
                        Longitude (WGS-84)
                        units :
                        degrees east
                        [1589760 values with dtype=float64]
                      • lon
                        (downtrack, crosstrack)
                        float64
                        ...
                        long_name :
                        Latitude (WGS-84)
                        units :
                        degrees north
                        [1589760 values with dtype=float64]
                      • elev
                        (downtrack, crosstrack)
                        float64
                        ...
                        long_name :
                        Surface Elevation
                        units :
                        m
                        [1589760 values with dtype=float64]
                      • glt_x
                        (ortho_y, ortho_x)
                        float64
                        ...
                        long_name :
                        GLT Sample Lookup
                        units :
                        pixel location
                        [4727177 values with dtype=float64]
                      • glt_y
                        (ortho_y, ortho_x)
                        float64
                        ...
                        long_name :
                        GLT Line Lookup
                        units :
                        pixel location
                        [4727177 values with dtype=float64]

                      We can see that each downtrack and crosstrack position has a latitude, longitude, and elevation, and the ortho_x and ortho_y data make up glt_x and glt_y arrays with a different shape. These arrays contain crosstrack and downtrack index values to quickly reproject the data. We will use these indexes to build an array of 2009x2353x285 (lat,lon,bands), filling it with the data from the reflectance dataset.

                      Go ahead and remove this dataset. We will use a function in the provided emit_tools module to orthorectify the data and place it into an xarray.Dataset.

                      In [17]:
                      del loc
del example

                      Import the emit_tools module and call use the help function to see how it can be used.

                      Note: This function currently works with L1B Radiance and L2A Reflectance Data.

                      In [18]:
                      import sys
sys.path.append('../modules/')
from emit_tools import emit_xarray
help(emit_xarray)

                      Help on function emit_xarray in module emit_tools:

emit_xarray(filepath, ortho=True, qmask=None, unpacked_bmask=None, GLT_NODATA_VALUE=0, fill_value=-9999)
    This function utilizes other functions in this module to streamline opening an EMIT dataset as an xarray.Dataset.
    
    Parameters:
    filepath: a filepath to an EMIT netCDF file
    ortho: True or False, whether to orthorectify the dataset or leave in crosstrack/downtrack coordinates.
    qmask: a numpy array output from the quality_mask function used to mask pixels based on quality flags selected in that function. Any non-orthorectified array with the proper crosstrack and downtrack dimensions can also be used.
    unpacked_bmask: a numpy array from  the band_mask function that can be used to mask band-specific pixels that have been interpolated.
    GLT_NODATA_VALUE: no data value for the GLT tables, 0 by default
    fill_value: the fill value for EMIT datasets, -9999 by default
                    
    Returns:
    out_xr: an xarray.Dataset constructed based on the parameters provided.

                      We can see that the emit_xarray function will automatically apply the GLT to orthorectify the data unless ortho = False. The function will also apply masks if desired during construction of the output xarray.Dataset. EMIT L2A Masks files provides a quality mask and a band_mask indicating if values were interpolated. For more about masking, see the How_to_use_EMIT_Quality_data.ipynb.

                      Use the emit_xarray function to read in and orthorectify the L2A reflectance data.

                      For a detailed walkthrough of the orthorectification process using the GLT see section 2 of the How_to_Orthorectify.ipynb in the how-tos folder.

                      In [19]:
                      ds_geo = emit_xarray(fp, ortho=True)
ds_geo
                      Out[19]:
                      <xarray.Dataset>
Dimensions:      (latitude: 2009, longitude: 2353, bands: 285)
Coordinates:
  * latitude     (latitude) float64 -39.31 -39.31 -39.31 ... -40.39 -40.4 -40.4
  * longitude    (longitude) float64 -62.51 -62.51 -62.51 ... -61.24 -61.24
    wavelengths  (bands) float32 ...
    fwhm         (bands) float32 ...
    spatial_ref  int64 0
Dimensions without coordinates: bands
Data variables:
    reflectance  (latitude, longitude, bands) float32 nan nan nan ... nan nan
Attributes: (12/38)
    ncei_template_version:             NCEI_NetCDF_Swath_Template_v2.0
    summary:                           The Earth Surface Mineral Dust Source ...
    keywords:                          Imaging Spectroscopy, minerals, EMIT, ...
    Conventions:                       CF-1.63
    sensor:                            EMIT (Earth Surface Mineral Dust Sourc...
    instrument:                        EMIT
    ...                                ...
    southernmost_latitude:             -40.39610428069674
    spatialResolution:                 0.000542232520256367
    spatial_ref:                       GEOGCS["WGS 84",DATUM["WGS_1984",SPHER...
    geotransform:                      [-6.25120945e+01  5.42232520e-04 -0.00...
    day_night_flag:                    Day
    title:                             EMIT L2A Surface Reflectance 60 m V001
                      xarray.Dataset
                        • latitude: 2009
                        • longitude: 2353
                        • bands: 285
                        • latitude
                          (latitude)
                          float64
                          -39.31 -39.31 ... -40.4 -40.4
                          long_name :
                          Latitude (WGS-84)
                          units :
                          degrees north
                          array([-39.306759, -39.307301, -39.307844, ..., -40.394478, -40.39502 ,
       -40.395562])
                        • longitude
                          (longitude)
                          float64
                          -62.51 -62.51 ... -61.24 -61.24
                          long_name :
                          Longitude (WGS-84)
                          units :
                          degrees east
                          array([-62.512095, -62.511552, -62.51101 , ..., -61.237848, -61.237306,
       -61.236764])
                        • wavelengths
                          (bands)
                          float32
                          ...
                          long_name :
                          Wavelength Centers
                          units :
                          nm
                          [285 values with dtype=float32]
                        • fwhm
                          (bands)
                          float32
                          ...
                          long_name :
                          Full Width at Half Max
                          units :
                          nm
                          [285 values with dtype=float32]
                        • spatial_ref
                          ()
                          int64
                          0
                          crs_wkt :
                          GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
                          semi_major_axis :
                          6378137.0
                          semi_minor_axis :
                          6356752.314245179
                          inverse_flattening :
                          298.257223563
                          reference_ellipsoid_name :
                          WGS 84
                          longitude_of_prime_meridian :
                          0.0
                          prime_meridian_name :
                          Greenwich
                          geographic_crs_name :
                          WGS 84
                          grid_mapping_name :
                          latitude_longitude
                          spatial_ref :
                          GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
                          array(0)
                        • reflectance
                          (latitude, longitude, bands)
                          float32
                          nan nan nan nan ... nan nan nan nan
                          long_name :
                          Surface Reflectance
                          units :
                          unitless
                          array([[[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
...
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]],

       [[nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        ...,
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan],
        [nan, nan, nan, ..., nan, nan, nan]]], dtype=float32)
                        • latitude
                          PandasIndex
                          PandasIndex(Float64Index([  -39.3067591475017, -39.307301380021954, -39.307843612542214,
               -39.30838584506247,  -39.30892807758273,  -39.30947031010298,
               -39.31001254262324, -39.310554775143494, -39.311097007663754,
               -39.31163924018401,
              ...
               -40.39068195549418, -40.391224188014434, -40.391766420534694,
               -40.39230865305495,  -40.39285088557521,  -40.39339311809546,
               -40.39393535061572,  -40.39447758313597,  -40.39501981565623,
               -40.39556204817649],
             dtype='float64', name='latitude', length=2009))
                        • longitude
                          PandasIndex
                          PandasIndex(Float64Index([  -62.5120945327963, -62.511552300276044, -62.511010067755784,
               -62.51046783523553,  -62.50992560271527,  -62.50938337019502,
               -62.50884113767476, -62.508298905154504, -62.507756672634244,
               -62.50721444011399,
              ...
               -61.24164373783563,  -61.24110150531537,  -61.24055927279512,
               -61.24001704027486, -61.239474807754604, -61.238932575234344,
               -61.23839034271409,  -61.23784811019384,  -61.23730587767358,
              -61.236763645153324],
             dtype='float64', name='longitude', length=2353))
                      • ncei_template_version :
                        NCEI_NetCDF_Swath_Template_v2.0
                        summary :
                        The Earth Surface Mineral Dust Source Investigation (EMIT) is an Earth Ventures-Instrument (EVI-4) Mission that maps the surface mineralogy of arid dust source regions via imaging spectroscopy in the visible and short-wave infrared (VSWIR). Installed on the International Space Station (ISS), the EMIT instrument is a Dyson imaging spectrometer that uses contiguous spectroscopic measurements from 410 to 2450 nm to resolve absoprtion features of iron oxides, clays, sulfates, carbonates, and other dust-forming minerals. During its one-year mission, EMIT will observe the sunlit Earth's dust source regions that occur within +/-52° latitude and produce maps of the source regions that can be used to improve forecasts of the role of mineral dust in the radiative forcing (warming or cooling) of the atmosphere.\n\nThis file contains L2A estimated surface reflectances and geolocation data. Reflectance estimates are created using an Optimal Estimation technique - see ATBD for details. Reflectance values are reported as fractions (relative to 1).
                        keywords :
                        Imaging Spectroscopy, minerals, EMIT, dust, radiative forcing
                        Conventions :
                        CF-1.63
                        sensor :
                        EMIT (Earth Surface Mineral Dust Source Investigation)
                        instrument :
                        EMIT
                        platform :
                        ISS
                        processing_version :
                        V1.0
                        institution :
                        NASA Jet Propulsion Laboratory/California Institute of Technology
                        license :
                        https://science.nasa.gov/earth-science/earth-science-data/data-information-policy/
                        naming_authority :
                        LPDAAC
                        date_created :
                        2022-11-14T09:50:54Z
                        keywords_vocabulary :
                        NASA Global Change Master Directory (GCMD) Science Keywords
                        stdname_vocabulary :
                        NetCDF Climate and Forecast (CF) Metadata Convention
                        creator_name :
                        Jet Propulsion Laboratory/California Institute of Technology
                        creator_email :
                        sarah.r.lundeen@jpl.nasa.gov
                        creator_url :
                        https://earth.jpl.nasa.gov/emit/
                        project :
                        Earth Surface Mineral Dust Source Investigation
                        project_url :
                        https://emit.jpl.nasa.gov/
                        publisher_name :
                        USGS LPDAAC
                        publisher_url :
                        https://lpdaac.usgs.gov
                        publisher_email :
                        lpdaac@usgs.gov
                        identifier_product_doi_authority :
                        https://doi.org
                        flight_line :
                        emit20220903t163129_o24611_s000
                        time_coverage_start :
                        2022-09-03T16:31:29+0000
                        time_coverage_end :
                        2022-09-03T16:31:41+0000
                        software_build_version :
                        010603
                        product_version :
                        01
                        history :
                        PGE Input files: radiance_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_rdn_b0106_v01.img, pixel_locations_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_loc_b0106_v01.img, observation_parameters_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_obs_b0106_v01.img, surface_model_config=/beegfs/store/emit/ops/repos/emit-sds-l2a/surface/surface_20221020.json
                        easternmost_longitude :
                        -62.5120945327963
                        northernmost_latitude :
                        -39.3067591475017
                        westernmost_longitude :
                        -61.236221412633064
                        southernmost_latitude :
                        -40.39610428069674
                        spatialResolution :
                        0.000542232520256367
                        spatial_ref :
                        GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
                        geotransform :
                        [-6.25120945e+01 5.42232520e-04 -0.00000000e+00 -3.93067591e+01 -0.00000000e+00 -5.42232520e-04]
                        day_night_flag :
                        Day
                        title :
                        EMIT L2A Surface Reflectance 60 m V001

                      1.5 Plotting Data - Orthorectified¶

                      Now that the data has been orthorectified, plot the georeferenced dataset using the same single wavelength (850nm) as above. We can use the aspect = 'equal' option to preserve the square pixel dimensions. The rasterize = True will help save memory and reduces the size of this notebook. For higher quality outputs, this can be omitted.

                      In [20]:
                      b850 = np.nanargmin(abs(ds_geo['wavelengths'].values-850)) # Find band nearest to value of 850 nm (NIR)
ds_geo.sel(bands=b850).hvplot.image(cmap='viridis', aspect = 'equal', rasterize=True) +\
     ds.sel(bands=b850).hvplot.image(cmap='viridis', aspect = 'equal', rasterize=True)
                      Out[20]:

                      We an also plot the data against an imagery tile using the geo=True and tiles= parameters instead of . Any tile source available in geoviews should work here. This will change the axis names, but that can be fixed by adding them manually in the options, like below.

                      In [21]:
                      ds_geo.sel(bands=b850).hvplot.image(cmap='viridis', frame_width=500, geo=True, tiles='EsriImagery',rasterize=True).opts(
    xlabel=f'{ds_geo.longitude.long_name} ({ds_geo.longitude.units})', ylabel=f'{ds_geo.latitude.long_name} ({ds_geo.latitude.units})')
                      Out[21]:

                      We can see that the orthorectification step placed the data on a geogrpahic grid that matches pretty well with ESRI tiles. Now that we have a better idea of what the target area looks like, we can also plot the spectra using the georeferenced data. First, filter out the water absorption bands like we did earlier.

                      In [22]:
                      ds_geo['reflectance'].data[ds_geo['reflectance'].data ==-0.01] = np.nan
# In newer versions of the file, this line below that takes advantage of the `good_wavelengths` data in the `sensor_band_parameters` group.
#ds_geo['reflectance'].data[ds_geo['good_wavelengths'].data==0] = np.nan

                      Now, plot the spectra at the Lat/Lon coordinates provided below.

                      In [23]:
                      point = ds_geo.sel(longitude=-61.833,latitude=-39.710,method='nearest')
point.hvplot.line(y='reflectance',x='wavelengths', color='black', frame_width=400).opts(
    title = f'Latitude = {point.latitude.values.round(3)}, Longitude = {point.longitude.values.round(3)}')
                      Out[23]:

                      1.6 Writing an Orthorectified Output¶

                      At this point, the ds_geo orthorectified EMIT data can also be written as a flattened netCDF4 output that can be read using the xarray.open_dataset function, if desired.

                      In [24]:
                      #ds_geo.to_netcdf('../data/geo_ds_out.nc')

# Example for Opening 
# ds = xr.open_dataset('../data/geo_ds_out.nc')

                      1.7 Interactive Spatial and Spectral Plots¶

                      Combining the Spatial and Spectral information into a single visualization can be a powerful tool for exploring and inspecting imaging spectroscopy data. Using the streams module from Holoviews we can link a spatial map to a plot of spectra.

                      We could plot a single band image as we previously have, but using a multiband image, like an RGB may help infer what targets we're examining. Build an RGB image following the steps below.

                      Select bands to represent red, green, and blue by finding the nearest to a wavelength chosen to represent that color.

                      In [25]:
                      # Find Nearest Bands
b650 = np.nanargmin(abs(ds_geo['wavelengths'].values-650)) # Find band nearest to value of 650 nm (red)
b560 = np.nanargmin(abs(ds_geo['wavelengths'].values-560)) # Find band nearest to value of 560 nm (green)
b470 = np.nanargmin(abs(ds_geo['wavelengths'].values-470)) # Find band nearest to value of 470 nm (blue)

                      Next, write a function to build an array from the chosen bands and scale the values using a gamma correction. Without applying this scaling the majority of the image would be very dark, with the reflectance data being skewed by the few pixels with very high reflectance.

                      Note: This has no impact on analysis or data, just visualizing the RGB map.

                      In [26]:
                      def gamma_adjust(ds,band):
    # Define Reflectance Array
    array = ds['reflectance'].sel(bands=band).data
    # Rescale Values using gamma to adjust brightness
    gamma = math.log(0.2)/math.log(np.nanmean(array)) # Create exponent for gamma scaling - can be adjusted by changing 0.2 
    scaled = np.power(array,gamma).clip(0,1) # Apply scaling and clip to 0-1 range
    scaled = np.nan_to_num(scaled, nan = 1) #Assign NA's to 1 so they appear white in plots
    return scaled

                      Now apply this function to each of the selected bands, stack them, build the arrays of coordinates (Lat, Lon, Bands) needed to create an xarray.Dataset, then build the dataset.

                      In [27]:
                      # Scale the Bands
r = gamma_adjust(ds_geo,b650)
g = gamma_adjust(ds_geo,b560)
b = gamma_adjust(ds_geo,b470)
# Stack Bands and make an index
rgb = np.stack([r,g,b]) # Stack r,g,b arrays and assign NA's to 1 so they appear white in plots
bds = np.array([0,1,2])
# Pull lat and lon values from geocorrected arrays
x = ds_geo['longitude'].values
y = ds_geo['latitude'].values
# Create new rgb xarray data array.
data_vars = {'RGB':(['bands','latitude','longitude'], rgb)} 
coords = {'bands':(['bands'],bds), 'latitude':(['latitude'],y), 'longitude':(['longitude'],x)}
attrs = ds_geo.attrs
ds_rgb = xr.Dataset(data_vars=data_vars, coords=coords, attrs=attrs)
ds_rgb.coords['latitude'].attrs = ds_geo['longitude'].attrs
ds_rgb.coords['longitude'].attrs = ds_geo['latitude'].attrs
ds_rgb
                      Out[27]:
                      <xarray.Dataset>
Dimensions:    (bands: 3, latitude: 2009, longitude: 2353)
Coordinates:
  * bands      (bands) int64 0 1 2
  * latitude   (latitude) float64 -39.31 -39.31 -39.31 ... -40.39 -40.4 -40.4
  * longitude  (longitude) float64 -62.51 -62.51 -62.51 ... -61.24 -61.24 -61.24
Data variables:
    RGB        (bands, latitude, longitude) float32 1.0 1.0 1.0 ... 1.0 1.0 1.0
Attributes: (12/38)
    ncei_template_version:             NCEI_NetCDF_Swath_Template_v2.0
    summary:                           The Earth Surface Mineral Dust Source ...
    keywords:                          Imaging Spectroscopy, minerals, EMIT, ...
    Conventions:                       CF-1.63
    sensor:                            EMIT (Earth Surface Mineral Dust Sourc...
    instrument:                        EMIT
    ...                                ...
    southernmost_latitude:             -40.39610428069674
    spatialResolution:                 0.000542232520256367
    spatial_ref:                       GEOGCS["WGS 84",DATUM["WGS_1984",SPHER...
    geotransform:                      [-6.25120945e+01  5.42232520e-04 -0.00...
    day_night_flag:                    Day
    title:                             EMIT L2A Surface Reflectance 60 m V001
                      xarray.Dataset
                        • bands: 3
                        • latitude: 2009
                        • longitude: 2353
                        • bands
                          (bands)
                          int64
                          0 1 2
                          array([0, 1, 2])
                        • latitude
                          (latitude)
                          float64
                          -39.31 -39.31 ... -40.4 -40.4
                          long_name :
                          Longitude (WGS-84)
                          units :
                          degrees east
                          array([-39.306759, -39.307301, -39.307844, ..., -40.394478, -40.39502 ,
       -40.395562])
                        • longitude
                          (longitude)
                          float64
                          -62.51 -62.51 ... -61.24 -61.24
                          long_name :
                          Latitude (WGS-84)
                          units :
                          degrees north
                          array([-62.512095, -62.511552, -62.51101 , ..., -61.237848, -61.237306,
       -61.236764])
                        • RGB
                          (bands, latitude, longitude)
                          float32
                          1.0 1.0 1.0 1.0 ... 1.0 1.0 1.0 1.0
                          array([[[1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.],
        ...,
        [1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.]],

       [[1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.],
        ...,
        [1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.]],

       [[1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.],
        ...,
        [1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.],
        [1., 1., 1., ..., 1., 1., 1.]]], dtype=float32)
                        • bands
                          PandasIndex
                          PandasIndex(Int64Index([0, 1, 2], dtype='int64', name='bands'))
                        • latitude
                          PandasIndex
                          PandasIndex(Float64Index([  -39.3067591475017, -39.307301380021954, -39.307843612542214,
               -39.30838584506247,  -39.30892807758273,  -39.30947031010298,
               -39.31001254262324, -39.310554775143494, -39.311097007663754,
               -39.31163924018401,
              ...
               -40.39068195549418, -40.391224188014434, -40.391766420534694,
               -40.39230865305495,  -40.39285088557521,  -40.39339311809546,
               -40.39393535061572,  -40.39447758313597,  -40.39501981565623,
               -40.39556204817649],
             dtype='float64', name='latitude', length=2009))
                        • longitude
                          PandasIndex
                          PandasIndex(Float64Index([  -62.5120945327963, -62.511552300276044, -62.511010067755784,
               -62.51046783523553,  -62.50992560271527,  -62.50938337019502,
               -62.50884113767476, -62.508298905154504, -62.507756672634244,
               -62.50721444011399,
              ...
               -61.24164373783563,  -61.24110150531537,  -61.24055927279512,
               -61.24001704027486, -61.239474807754604, -61.238932575234344,
               -61.23839034271409,  -61.23784811019384,  -61.23730587767358,
              -61.236763645153324],
             dtype='float64', name='longitude', length=2353))
                      • ncei_template_version :
                        NCEI_NetCDF_Swath_Template_v2.0
                        summary :
                        The Earth Surface Mineral Dust Source Investigation (EMIT) is an Earth Ventures-Instrument (EVI-4) Mission that maps the surface mineralogy of arid dust source regions via imaging spectroscopy in the visible and short-wave infrared (VSWIR). Installed on the International Space Station (ISS), the EMIT instrument is a Dyson imaging spectrometer that uses contiguous spectroscopic measurements from 410 to 2450 nm to resolve absoprtion features of iron oxides, clays, sulfates, carbonates, and other dust-forming minerals. During its one-year mission, EMIT will observe the sunlit Earth's dust source regions that occur within +/-52° latitude and produce maps of the source regions that can be used to improve forecasts of the role of mineral dust in the radiative forcing (warming or cooling) of the atmosphere.\n\nThis file contains L2A estimated surface reflectances and geolocation data. Reflectance estimates are created using an Optimal Estimation technique - see ATBD for details. Reflectance values are reported as fractions (relative to 1).
                        keywords :
                        Imaging Spectroscopy, minerals, EMIT, dust, radiative forcing
                        Conventions :
                        CF-1.63
                        sensor :
                        EMIT (Earth Surface Mineral Dust Source Investigation)
                        instrument :
                        EMIT
                        platform :
                        ISS
                        processing_version :
                        V1.0
                        institution :
                        NASA Jet Propulsion Laboratory/California Institute of Technology
                        license :
                        https://science.nasa.gov/earth-science/earth-science-data/data-information-policy/
                        naming_authority :
                        LPDAAC
                        date_created :
                        2022-11-14T09:50:54Z
                        keywords_vocabulary :
                        NASA Global Change Master Directory (GCMD) Science Keywords
                        stdname_vocabulary :
                        NetCDF Climate and Forecast (CF) Metadata Convention
                        creator_name :
                        Jet Propulsion Laboratory/California Institute of Technology
                        creator_email :
                        sarah.r.lundeen@jpl.nasa.gov
                        creator_url :
                        https://earth.jpl.nasa.gov/emit/
                        project :
                        Earth Surface Mineral Dust Source Investigation
                        project_url :
                        https://emit.jpl.nasa.gov/
                        publisher_name :
                        USGS LPDAAC
                        publisher_url :
                        https://lpdaac.usgs.gov
                        publisher_email :
                        lpdaac@usgs.gov
                        identifier_product_doi_authority :
                        https://doi.org
                        flight_line :
                        emit20220903t163129_o24611_s000
                        time_coverage_start :
                        2022-09-03T16:31:29+0000
                        time_coverage_end :
                        2022-09-03T16:31:41+0000
                        software_build_version :
                        010603
                        product_version :
                        01
                        history :
                        PGE Input files: radiance_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_rdn_b0106_v01.img, pixel_locations_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_loc_b0106_v01.img, observation_parameters_file=/beegfs/store/emit/ops/data/acquisitions/20220903/emit20220903t163129/l1b/emit20220903t163129_o24611_s000_l1b_obs_b0106_v01.img, surface_model_config=/beegfs/store/emit/ops/repos/emit-sds-l2a/surface/surface_20221020.json
                        easternmost_longitude :
                        -62.5120945327963
                        northernmost_latitude :
                        -39.3067591475017
                        westernmost_longitude :
                        -61.236221412633064
                        southernmost_latitude :
                        -40.39610428069674
                        spatialResolution :
                        0.000542232520256367
                        spatial_ref :
                        GEOGCS["WGS 84",DATUM["WGS_1984",SPHEROID["WGS 84",6378137,298.257223563,AUTHORITY["EPSG","7030"]],AUTHORITY["EPSG","6326"]],PRIMEM["Greenwich",0,AUTHORITY["EPSG","8901"]],UNIT["degree",0.0174532925199433,AUTHORITY["EPSG","9122"]],AXIS["Latitude",NORTH],AXIS["Longitude",EAST],AUTHORITY["EPSG","4326"]]
                        geotransform :
                        [-6.25120945e+01 5.42232520e-04 -0.00000000e+00 -3.93067591e+01 -0.00000000e+00 -5.42232520e-04]
                        day_night_flag :
                        Day
                        title :
                        EMIT L2A Surface Reflectance 60 m V001

                      Now that we have an RGB dataset, use it to build our spatial plot.

                      In [28]:
                      # Define RGB Image
map = ds_rgb.hvplot.rgb(x='longitude', y='latitude', bands='bands', aspect = 'equal', frame_width=400)

                      To visualize the spectral and spatial data side-by-side, we use the pointerXY and Tap functions from the streams module from the holoviews library.

                      Define objects resulting from the stream of the pointer x and y position on our spatial plot, map, then define objects resulting from a clicked x and y position on the same map.

                      Next, build a function to plot the spectra based on these two sets of x and y coordinates on the map. This will allow us to return spectra from a position we clicked on the image, and spectra where the mouse is currently hovering, allowing comparison of pixel reflectance values.

                      In [29]:
                      # Stream of X and Y positional data
posxy = hv.streams.PointerXY(source=map, x=-61.833, y=-39.710) 
clickxy = hv.streams.Tap(source=map, x=-61.833, y=-39.710) 

# Function to build a new spectral plot based on mouse hover positional information retrieved from the RGB image using our full reflectance dataset 
def point_spectra(x,y):
    return ds_geo.sel(longitude=x,latitude=y,method='nearest').hvplot.line(y='reflectance',x='wavelengths',
                                                                           color='#1b9e77', frame_width=400)
# Function to build spectral plot of clicked location to show on hover stream plot
def click_spectra(x,y):
    clicked = ds_geo.sel(longitude=x,latitude=y,method='nearest')
    return clicked.hvplot.line(y='reflectance',x='wavelengths', color='black', frame_width=400).opts(
        title = f'Latitude = {clicked.latitude.values.round(3)}, Longitude = {clicked.longitude.values.round(3)}')
# Define the Dynamic Maps
point_dmap = hv.DynamicMap(point_spectra, streams=[posxy])
click_dmap = hv.DynamicMap(click_spectra, streams=[clickxy])

# Plot the Map and Dynamic Map side by side
(map + click_dmap*point_dmap)
                      Out[29]:

                      Contact Info:¶

                      Email: LPDAAC@usgs.gov
                      Voice: +1-866-573-3222
                      Organization: Land Processes Distributed Active Archive Center (LP DAAC)¹
                      Website: https://lpdaac.usgs.gov/
                      Date last modified: 02-02-2022

                      ¹Work performed under USGS contract G15PD00467 for NASA contract NNG14HH33I.